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ABSTRACT 


This study addresses the issues concerning the upgrade and reuse of computer 
simulation models and presents a comprehensive methodology — The Fidelity 
Enhancement Process — for conducting a model upgrade. Recent advances in software 
technology — specifically object-oriented programming and open architecture system 
development — have made this process feasible and provide unprecedented 
opportunities for model reuse. The Fidelity Enhancement Process was developed and 
applied to the Marine Corps Communication Architecture Analysis Model (MCCAAM) 
during its upgrade. MCCAAM simulates Marine Air Ground Task Force (MAGTF) 
single-channel communications architectures. MCCAAM was modified to evaluate 
architecture performance under different allocations of next-generation radios to units 


in the MAGTF, where the performance of an allocation was tactically driven. 
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I. INTRODUCTION 


A. STATEMENT OF THE PROBLEM 

In the current state of computer simulation, a model’s ability to perfectly 
replicate the system being modelled is limited by time: a hardware based constraint. 
This limitation is continuously being reduced by advancing technology, which provides 
ever greater computing speed and quicker memory access. At the same time, software 
design improvements have given us the ability to modify and reuse existing models to 
meet new requirements. 

Combined, these capabilities place great power in the hands of the analyst and 
give him a wide range of options for improved model design and expanded model 
usage. Of special interest is the ability that this increased power gives the analyst to 
upgrade existing models — to make good models even better. The big question 
becomes: how do we determine where best to apply our expanded capability to achieve 
this goal? 

It seems appropriate to focus on areas of the model that were previously limited 
or ignored because of hardware or software constraints. It also makes sense to update 
the model to reflect any changes in the real world system that it represents; this may 
require changing the model’s parameters or structure. The exploration of new model 
uses may also merit some of the available power. How do we choose from among these 
alternatives? The extent of the improvements to apply to each submodel is still 
another complicated decision. It appears that all of these issues must be addressed to 


effectively exploit the available power. The need for a coherent, scientific method to 











choose which submodels to upgrade is apparent, but there is no methodology available 


today that addresses these issues in an organized manner. 


B. PURPOSE AND SCOPE 

The purpose of this thesis is to develop a structured methodology for upgrading 
an existing model. The importance of such a methodology can be measured in terms 
of the time and money saved by not developing new models. The methodology 
concentrates on the questions posed earlier including: which parts of the model should 
be improved? and how much improvement does each part need? 

In scope, this thesis is limited to developing a generic methodology and applying 
it to upgrade a single model — the Marine Corps Communication Architecture Analysis 
Model (MCCAAM) — which was developed at the Naval Postgraduate School by a team 
of analysts, including the author. The model simulates Marine Air Ground Task Force 
(MAGTF) single-channel communications architectures. MCCAAM was modified to 
evaluate architecture performance under different allocations of next-generation radios 


to units in the MAGTF, where the performance of an allocation was tactically driven. 


C. APPROACH 

New software technology — specifically object-oriented programming and open 
architecture (both described below) — have provided unprecedented opportunities for 
model reuse. It has become easier (and perhaps cheaper) to build and try model 
improvements than to determine a priori which improvements are worth pursuing. 
This thesis seeks to exploit these software innovations by presenting an organized 


methodology for identifying the model enhancements that might be worthwhile, 





building and evaluating prototypes of these enhancements and retaining only those 


deemed worthy. 








Il. BACKGROUND 


A. DEFINITIONS 

Before exploring the background material, some definitions are provided to bind 
the concepts and ideas that follow. 

A model’s level of fidelity is the degree to which the model produces the same 
outcomes as the tangible physical system it represents. Therefore a model with 
infinite fidelity would produce results identical to those of the actual system. 

Aggregation is the extent to which a group of things in the real world have 
been consolidated in the model. A model that depicts an army in conflict as a single 
entity (object) is highly aggregated, whereas the depiction of 500,000 unique soldier 
objects represents total disaggregation. 

Model resolution is the degree to which submodels are disaggregated. 
Resolution is the generic level of disaggregation within a submodel. Increasing 
resolution means replacing simple decision logic with more complex logic, using more 
source data, including more objects, or simply improving approximations at the cost 
of computational performance. Generally, a high resolution model also has high 


fidelity. 


B. OBJECT ORIENTED SIMULATION 
Object oriented simulation (OOS) provides a rich and easily understood 
environment for building computer models of real world systems. MCCAAM was 


written in MODSIM II, a general purpose, modular, high-level programming language 











that provides direct support for object-oriented programmiug[Ref. 1]. The 
following discussion of object oriented simulation uses MODSIM II terminology. 

The modular structure of OOS directly supports model reuse by allowing 
programs to be constructed from library modules. Each library module contains a 
Definition Module outlining the type declarations and an Implementation Module 
that includes all of the executable code for the methods and procedures. This 
structure enables the developer to easily improve or modify one module with minimal 
impact on the remainder of the model and minimal additional development costs. The 
modular design of library modules and the objects they describe provide tremendous 
potential for comprehension and eventual reuse by the user as well as the designer. 

An "object" combines a data record, which describes the state of the object, with 
procedures that describe its behaviors[Ref. 1]. The procedures are discussed first. 
They are called methods, and they describe the actions that the object can perform. 
The ASK METHOD in MODSIM II is equivalent to a procedure call in most 
languages: the actions are executed immediately, without passing any simulation time. 
The TELL METHOD, on the other hand, is executed asynchronously: the simulation 
continues for some time after a TELL METHOD has been called and during its 
execution. This allows a TELL METHOD to pass simulation time as a part of its 
actions. While the object’s methods represent its actions, its state is reflected by its 
fields. 

An object’s fields are much like those of a normal record structure except that 
they can only be modified by the object’s own methods. This enables the model 
developer to exercise total control over the changes made in these fields. Problems 


become easier to identify since they must be in that object’s own methods. Although 








these values can not be changed by other objects, they can be "read" by other parts of 
the program. 

This reference to other parts of the program brings up questions about the 
structure of the program. The single object described above is merely an object type 
with specified fields and methods. The object type serves as a template or 
specification. Object instances are created from it, dynamically, during the 
simulation. Once an object instance is created, its methods can be invoked by 
messages from other objects that ask it to perform its methods. 

After an object type has been defined by its library modules, new types can be 
evolved from it. Each descendent in the resulting hierarchy can add its own fields and 
methods to those of its ancestors or modify an inherited method. Thus, if we take a 
collection of objects which share Vehicle Object as their ancestor and ask each to 
refuel, the Car Object might take on unleaded gas, the Truck Object diesel fuel and 
the Mule Object would eat hay[Ref. 1]. The capability of performing different actions 
with the same command is referred to as polymorphism. Combined with inheritance, 


it forms a solid foundation for reusing these object types. 


C. OPEN ARCHITECTURE 

In addition to OOS, a second major software technological advance is the move 
toward open architecture system development. Because of the new degree of 
standardization produced by open architecture, models are portable between 
computing architectures to an unprecedented degree. The term open architecture 
implies that some degree of standardization has been achieved in 


®@ operating systems, 





@ graphical user interfaces, 

e data base management interfaces, 

e network operations and protocols, and 

@ interfaces to presentation graphics programs. 
Open architecture provides the potential for model migration to improve performance 
or to realize any necessary capability upgrades. Reimplementing existing models on 
new architectures will no longer require developers to change the model’s code. This 
alone represents a tremendous savings in simulation effort that can be applied to 
upgrading existing models rather than recoding models to _ support 


migration.[Ref. 2] 


D. MCCAAM 

The Marine Corps Communication Architecture Analysis Model (MCCAAM) 
simulates Marine Air Ground Task Force (MAGTF) FM single-channel radio 
communication architectures. The model uses a workload paradigm of Marine Broad 
Operational Tasks (MBOTS), Broad Operational Subtasks (BOSTs) and Message 
Exchange Occurrences (MEOs). This framework has been fitted to all of the standard 
message traffic within the Marine Corps[Ref. 3]. 

An MBOT encompasses a broad mission area and contains related tasks such as 
the MBOT Artillery Call For Fire. Each MBOT is further broken down into BOSTs, 
which represent specific tasks that are executed by units of specific types. For 


example, the Standard Call For Fire is one of the BOSTs contained in the MBOT 


Artillery Call For Fire; it is initiated by a Battery Forward Observer. Each BOST is 
made up of a set of precedence constrained communications requirements, its MEOs. 

An MEO specifies the unit types of the receivers as well as the net type used for 
its transmission. The first MEO of the Standard Call for Fire is a transmission from 
the Forward Observer to the Battalion Fire Direction Center on the Battalion’s 
Conduct of Fire net. This traffic structure allows the model to generate realistic, 


interdependent message traffic.[Ref. 4] 


E. RISK MANAGEMENT 

The early identification of risk areas is crucial to the successful completion of any 
software development effort. Risk areas encompass logic, algorithms, data, and their 
associated assumptions. 

The importance of risk management to military modelling is documented in the 
DoD Standard on Software Development, which calls for the documentation and 
implementation of procedures for risk management([Ref. 5). The Risk 
Management Plan provides a useful framework for overcoming major sources of 
program risk. 

A Risk Management Plan ensures that each project makes early identification 
of its top risk areas. These risk areas include potential cost and schedule problems as 
well as the technical risks mentioned earlier. It is important to develop a stratezy for 
resolving these risk areas early in the development process. In addition, continuing 
emphasis should be maintained through periodic reviews and the resolution of new 
risk areas as they surface. Proper use of risk management will ensure the appropriate 


focus on early prototyping, simulation, key personnel staffing measures and other risk 


resolving techniques. This risk-driven approach helps the developer avoid problems 


that might otherwise jeopardize a successful model upgrade[Ref. 6]. 














Ill. FORMULATION OF THE PROCESS 


A. THE FIDELITY ENHANCEMENT PROCESS 

The Fidelity Enhancement Process is a risk-driven approach to increasing the 
resolution of an existing simulation model. The use of OOS and open architecture 
provide the flexibility needed by the developer to efficiently upgrade an existing model 
with this process. As a result, the process is directed primarily toward models that 
have been implemented in OOS environments that support open architecture. 

Models that do not meet these criteria present limited opportunities for reuse. 
The model’s lack of flexibility is detrimental to its reuse and may preclude any 
upgrade whatsoever. In fact, reimplementing these models in the desired format may 
not be possible due to the general incompatibility between most high level languages 
and object oriented programming languages. Compatibility problems beiween differing 
OOS environments may also preclude language changes. However, incorporating a 
newer version of the current programming language or a compatible graphics package 
are valid changes that can be implemented. 

The five stage Fidelity Enhancement Process is a comprehensive methodology 
for upgrading existing computer simulation models. It is formulated for simulations 
that produce a decision from a finite set of alternatives. The stages are executed 


consecutively as portrayed in Figure 1. 
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Figure 1 The Fidelity Enhancement Process 


B. STAGE 1 -THE MODEL ASSESSMENT 

The model assessment stage establishes the foundation and limits of the fidelity 
enhancement. The current capabilities of the model make up its foundation, and the 
circumstances which motivate the upgrade establish the limits. These limits are 
either hardware-driven, model-driven or a combination of both. Before exploring the 


limits, it is important that we ensure the foundation is sound. 
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The first step of the model assessment updates the risk areas within the current 
model. Risk areas encompass logic, algorithms, data, and their associated assumptions. 
These risk areas were used by the model developer to justify the model’s current level 
of resolution. Although they were acceptable when the model was delivered, new data, 
requirements or standards may invalidate key assumptions that were made or logic 
that was used. Any discrepancies must be addressed at this point to ensure a strong 
foundation prior to setting the model upgrade limits. 

With the foundation in place, the model upgrade limits are determined by 
analyzing the events which generated the need for a better model. Improvements in 
the computer hardware used by the model are the primary force behind a hardware- 
driven upgrade. In this case, the user’s primary goal is to effectively utilize the 
increased capability. Closely related is the desire to migrate the model to a larger 
machine, such as a move from a PC to a workstation. In either case, the hardware 
issue becomes one of known dimensions, which are specified by the end user. 

Model-driven upgrades involve the addition of specific capabilities or the 
enhancement of existing capabilities. This case is the most likely scenario, because 
historically simulation models have focused on specific problems under specific 
conditions. The need to solve a related problem under the same or changed conditions 
presents an opportunity for a model-driven upgrade. The limits of this upgrade may 
encompass both software and hardware issues. 

This combination emphasizes the flexibility of open-architecture. Although the 
hardware currently supporting the model may offer some potential for expansion, the 


option may exist to migrate to a more capable machine. This combined option 
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presents the opportunity to maximize the number of potential enhancements while 


holding down any resultant degradation of model performance. 


C. STAGE 2 - FIDELITY ENHANCEMENT REQUIREMENTS 
Requirements development begins with the definition of possible upgrade 
requirements. These represent both the end-user’s "wish list” and the developer’s 
vision of the next version of his model. Both may include new or modified user 
interface requirements as well as additional model capabilities. These needs are 
consolidated into a requirements list, which is developed jointly by the end user and 
developer. It includes all of the proposed model enhancements and their risk areas. 
Once all the requirements have been identified, the steps required to implement 
each of them are outlined by the developer. They include the changes to the model 
for each requirement and their impact on the associated risk areas. This information 
is used by the developer as he formulates the specific modifications needed to add the 
proposed enhancements to the existing model. These modifications may include the 
addition of new modules or objects to the model as well as the modification of existing 


modules. 


D. STAGE 3 - PROTOTYPING 

The fundamental benefit provided by this model upgrading process is the ability 
to incorporate enhancements while minimizing the changes required to the current 
model. This integration of enhancements is accomplished through prototyping. 


Although the word "prototype" brings to mind the experimental version of a system 
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used during preliminary design work, its potential as a tool for fidelity enhancement 
goes well beyond that limited view. 

The prototyping necessary for the Fidelity Enhancement Process is strawman 
prototyping, which provides a surrogate system that can be investigated to improve 
the design of the eventual upgraded system[Ref. 7]. Each enhancement on the 
requirements list is integrated into the existing model in such a way that it can be 
turned on and off with software switches. This enables the developer to assess the 


impact of each enhancement combination on the model’s overall performance. 


E. STAGE 4 - FIDELITY ANALYSIS 
Once the prototyped enhancements are in place, the developer must assess the 
costs and benefits of his new enhanced model. The underlying assumption of fidelity 
analysis is that by increasing model resolution, the model’s fidelity must also increase. 
While the result of increased model fidelity is a better model providing better answers, 
the costs incurred by fidelity enhancement must also be addressed. 
1. Fidelity Costs 
Any increase in model fidelity produces a corresponding decrement to the 
execution of the model in terms of computing speed, the amount and types of data 
required and model sophistication. These decrements represent the fidelity costs 
inherent in the fidelity enhancement process. Other than the relatively low software 
modification costs, what exactly are these fidelity costs? 
a. Performance Degradation 


It is easy to predict that the enhanced model will experience longer 


execution times if it is run on the current hardware. But, the improved speed of 








newer hardware may compensate for the longer execution times. Therefore, model 
migration is one option that should be considered. A move to a more capable machine 
may be necessary to realize the needed enhancements. However, this move must be 
approved in advance by the end user to verify his ability to support the new hardware 
requirements. The developer must then test the enhanced model on the proposed 
platform and budget his expanded capability accordingly. 
b. Model Sophistication 

To increase the detail of a submodel, the developer increases the 
required level of understanding for himself and the user. Although the user has the 
option of viewing the submodel as a "black box", the acceptance and confidence in the 
answers rendered will normally necessitate the user’s comprehension and 
understanding of the model’s risk areas. His ability to effectively utilize an enhanced 
user interface may also depend on thorough understanding of the model’s internal 
processes. The developer, on the other hand, must be an expert. His expertise should 
encompass the physical system being modelled as well as the model itself. This 
knowledye is crucial to the definition and application of the appropriate parameters 
to the model. Any shortcomings in this area also increase data risk. 

c. Data Risk 

Data risk derives from the effects of increased resolution on the data 
required to run the enhanced model. One effect is the need for additional data to 
support the greater level of detail being modelled. Another effect is increased 
sensitivity to the accuracy of the data. The developer and user are often called upon 


to estimate the parameters or even the probability distributions used by the model. 
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The degree of confidence (or lack thereof) in the user’s ability to obtain the data and 
the quality of the sources make up an enhancement’s data risk. 

Unlike the first two fidelity costs, certain data risks can be addressed 
through sensitivity analysis. The ability to quantify these risks or demonstrate their 
limited impact may alleviate their costs. This is accomplished by varying the 
enhancement’s parameters to gauge the model’s response. It may also be useful to 
test each enhancement with varied parameters. Each variant would be tested as a 
separate enhancement. These related variants would then be compared to determine 
the sensitivity of the enhancement to its parameters. This analysis would take place 
in conjunction with the determination of the fidelity benefits. 

2. Fidelity Benefits 

The fidelity benefits from individual enhancements manifest themselves 
as incrementally better answers to the questions being asked or choices being made. 
This higher resolution level may raise the end-user’s confidence in, and acceptance of, 
the model’s decisions. Depending on the upgrade involved, the model may answer 
new, more detailed questions or provide more detailed answers to existing questions. 
Another area that may benefit from the upgrade is the accuracy of the decision 
rendered. These possible effects are dependent on the particular model and seem 
difficult to quantify. But the ability to quantify the fidelity benefit or yield of each 
enhancement is crucial to the Fidelity Enhancement Process. This quantification is 


the focus of the fidelity assessment. 
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3. Fidelity Assessment 

The fidelity assessment is the cornerstone of the Fidelity Enhancement 
Process. It encompasses the collection and processing of all the fidelity costs and 
benefits. The assessment begins with the establishment of the test case, which 
includes the selection of the data sets necessary to run the model. Each of these data 
sets contain one of the model’s alternatives, which is a possible solution that is 
compared to all of the other possible solutions to determine the best. Once the set of 
possible alternatives have been identified, the next step is to establish the decision 
boundaries. 

The decision boundaries are established in terms of the baseline and 
topline cases. Each of these cases represent an upgrade combination: a combination 
of enhancements that are turned "on" and "off". The baseline case corresponds to no 
enhancements turned "on". This case produces an answer equivalent to that of the 
original model. The decision produced by switching all of the enhancements "on" is 
the topline case. This case represents the decision produced at maximum resolution. 
The first question to be answered is: are the baseline and topline decisions equal? If 
they are, the enhancements have produced an insignificant increase in model fidelity 
and should be left out of the model, avoiding their fidelity costs. If the decisions 
differ, the fidelity analysis continues with the determination of a weight for each 
alternative. 

The model’s decision output from the topline case is utilized to construct 
the weights used by the Measure of Performance (MOP) equation (below). A weight 


is determined for each of the alternatives outlined earlier. Each alternative’s weight 
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corresponds to the proportion of times it was chosen as the best alternative by the 


topline case. 


Alternative number 
Weight of Alternative j 


= Number of Alternative j decisions 
Number of Replications 


OHM. 
U 


The MOP can then be calculated for each proposed upgrade combination by running 
R replications of the model and summing the products of the weights and the number 


of times each alternative wa: chosen. 


n 
wat. 2k 
MOP, = aoe Ay; W; 
J* 


Z = Upgrade Combination Number 
j = Alternative Number 
MOP,= Measure of Performance for Upgrade Combination i 


A;; = Number of Al ternative j Decisions 


W, = Weight of Alternative j 
R = Number of Replications 
n = Number of Alternatives 


The evaluation of these MOPs and the direction alung which the fidelity assessment 
proceeds become model dependent at this point. 

For an upgrade involving a relatively small number of enhancements, a 2" 
factorial design is preferred. This special case of general factorial design is keyed to 
the comparison of factors with only two levels. In this case, "k" represents the number 


of enhancements while the two levels are "on" and "off". This type of analysis allows 


18 











the analyst to more accurately gauge the interactions between the k factors and their 
effect on the model’s response[Ref. 8]. For upgrades that entail numerous 
enhancements, the number of experimental runs required for a factorial design may 
become prohibitive. Alternate designs, such as two-stage designs or single factor 
analysis are more appropriate for these situations. 

Single factor analysis treats each er Lancement as an alternative system and 
discounts the interactions between them. Common examples of this technique include 
randomized complete block design and other forms of one way analysis of variance[Ref. 
(Ref. 9}. Or, as an alternative, the best of a group of similar enhancements may 
be chosen using "two-stage" sampling. This technique also treats each enhancement 
as an individual alternative. The analyst estimates the variance produced during the 
first stage. This is used to establish the number of runs required for the second stage, 
which produces the final decision[Ref. 9]. A slight modification of this technique 
selects a fixed number of best enhancements from the total[Ref. 9]. This method can 
be used to trim down the number of enhancements to a level more conducive to a 
factorial design. This brief look provides some ideas concerning different possibilities 
for enhancement analysis. It remains the job of the analyst to tailor this assessment 
to the characteristics and needs of his model upgrade and provide the decision maker 


with the data needed to make an informed fidelity decision. 


F. STAGE 5 - FIDELITY DECISION 
The fidelity decision stage consolidates the analysis of fidelity benefits with that 


of the fidelity costs. The end user then compares the enhancement yields with their 
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associated performance decrements to the subjective analysis of model sophistication 


and data risk and makes his decision. 
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IV. APPLICATION OF THE PROCESS 


A. BACKGROUND 

The Fidelity Enhancement Process was applied to the Marine Corps 
Communication Architecture Analysis Model (MCCAAM). This proved beneficial even 
though MCCAAM was in its initial development. 

MCCAAM is a computer simulation of Marine Corps single-channel radio 
architectures[Ref. 4]. It replicates the interactions between units, radios and nets in 
a realistic manner using the BOST message structure explained in section II D. The 
network architecture is constructed dynamically from the input data, which results in 
nearly unlimited flexibility: the model can be applied to radio networks of all types and 
sizes. A penalty process gauges the number of BOSTs that are not completed in a 
timely manner. Each BOST has an allotted time for completion after which the 
architecture is immediately assessed a BOST-specific one time penalty and then a 
(again BOST-specific) constant penalty rate until it is completed. This penalty process 
is used to assess the performance of a given architecture in terms of its long term 
penalty rate. 

The original (baseline) model uses the internal penalty process to choose the best 
architecture from a finite set of alternatives. In addition to this capability, the 
problem of choosing the best allocation of new SINCGARS radios as partial 
replacements for current PRC-77 radios in an existing architecture was posed. This 
particular application of MCCAAM required enhancements that would differentiate 


between the two radio types. 
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B. STAGE 1 -THE MODEL ASSESSMENT 

The model assessment proceeded rapidly because the upgrade was executed 
almost concurrently with the model’s initial development. As a result, the risk areas 
were up to date, the model’s foundation was sound and the boundaries were 
established during the original development effort. 

The model’s hardware boundaries were dictated by the software memory 
limitations of MODSIM’s PC version and its C compiler. The model quickly outgrew 
that platform and was subsequently migrated to a SUN workstation with a newer 
version of MODSIM II. The previously encountered limitations were alleviated by this 


move, which was cleared by the end-user prior to its adoption by the developers. 


C. STAGE 2 - FIDELITY ENHANCEMENT REQUIREMENTS 

This model-driven upgrade resulted in the development of numerous 
enhancements including a new object, which portrayed the effects of enemy jamming 
systems, and changes to existing objects. 

1. Jammer Object 

The introduction of enemy jamming to the model was considered crucial 

because the newer SINCGARS radios have a frequency hopping capability that make 
them effectively "jam-proof", and this is the primary difference between the two radios. 
The new Jammer object type was developed as a generic specification that could be 
applied to any enemy jamming system. The parameters used to specify each jar.mer 


include its location and type as well as the jamming direction, range and duration. 
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2. Mean Time Between Failures 

An additional discriminator between the two radios is the expected mean 
time between failures (MTBF). Test and evaluation of the new radio reflects a 
significant increase in reliability for the SINCGARS. In addition, the modular design 
of the SINCGARS radio gives it a shorter mean repair time. However, once the 
SINCGARS radio is repaired or replaced, the process of rejoining a frequency-hopping 
SINCGARS net requires significantly more time than rejoining a PRC-77 single 
frequency net. As a result, parameters reflecting the MTBF, repair time, and net 
entry time for each radio type were estimated. The radio object and its methods 
required modification to incorporate these changes. 

3. Perishability 

The developers found that the need for action following the loss of a radio 
or access to a net induced a complicated series of events. Although the routing of 
traffic on a functional network is relatively straight forward, the alternate routing 
procedures required to deal with enemy jamming and equipment failure proved to be 
very complex. Incorporating an algorithm to accomplish the alternate routing required 
the addition of a completely new module. 

Bottlenecks formed in the network by the previously mentioned changes raised 
questions concerning the perishability of message traffic. A message that is trapped 
in an inoperational radio’s queue may reach a point where its transmission is no longer 
a valid requirement. At this point, the message is considered perishable and is 
removed from the network. The enhancements required to implement this capability 


included the modification of the BOST structure and changes to the BOST data files. 
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D. STAGE 3 - PROTOTYPING 

The enhancements described above were systematically added during model 
development. The software switches used to enable the enhancements could be 
switched on to work on the enhancements or off for unimpeded work on the baseline 
model. By adhering to this practice, the development of both proceeded with minimal 
conflict. 


BE. STAGE 4 - FIDELITY ANALYSIS 
Three enhancements were implemented to differentiate between SINCGARS 
radio allocations: Jammers, MTBF, and Perishability. 
1. Fidelity Costs 
The fidelity costs incurred by the chosen enhancements include 
performance degradation, model sophistication and data risk. Performance degradation 
was measured in terms of clock time. A stopwatch was used to measure the extra time 
required for each of the enhancements. In addition, the subjective costs were 
evaluated for each of the enhancements. 
a. dammer Object 
The addition of the jammer object introduced a great deal of model 
sophistication and data risk. The baseline model and its portrayal of the MAGTF radio 
network is comprehendible to anyone with experience in tactical operations. However, 
the proper application of electronic warfare to the model requires additional expertise 
on the part of the user. The choice of jammer type and employment strategy must be 
made during the creation of the jammer data file and prior to model execution. Data 


risk is generated by the choices previously mentioned as well as the jammer’s 
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parameters which, in this case, do not reflect terrain features or weather 
considerations. 
b. Radio Failures 
Although the sophistication required to utilize the radio failure 
enhancement is comparable to that required by the baseline model, there is some data 
risk involved. The actual MTBF is an estimated parameter as is the "repair or replace” 
time for each. Other data risk issues involve modelling the substitution of broken 
radios with spares or switching frequencies between nets on an operational radio. 
c. Message Perishability 
Message perishability is primarily data risk sensitive. These risks 
include a subjective judgement whether or not each BOST is perishable as well as the 
estimation of perishability points for those that are. 
2. Fidelity Benefits 
The fidelity benefits manifest themselves as better radio allocation 
decisions. 
3. Fidelity Assessment 
a. Selection of Alternatives 
The fidelity assessment began with the specification of the allocation 
alternatives. A sample data set was developed that focused on the Ground Combat 
Element (GCE) of a Marine Expeditionary Brigade (MEB). The units, nets and BOSTs 
that were stressed involved indirect fire support and tactical communications. Three 
different radio allocations were chosen as possible solutions to the optimal allocation 


problem. These three radio allocations served as the alternatives. 
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Table I Allocation Alternatives 


ALTERNATIVES 
# SINCGARS 
FEBA Back Configuration 


All PRC-77 Configuration 





ae 
| 2 | Top Down Configuration 


The first two alternatives reflect the possible tactical employment of 
these communications assets on the battlefield while the third represents the current 
architecture. The forward edge of the battle area (FEBA) back alternative places the 
SINCGARS radios on those nets that are physically closest to enemy jamming assets 
and carry the bulk of the architecture’s traffic load. These include nets of battalion 
level and lower. The top down alternative focuses the employment of SINCGARS 
radios in the nets controlled by higher headquarters. These nets normally have a 
greater number of subscribers and process the most important message traffic. The 
third alternative depicts the current tactical architecture with no SINCGARS radios 
employed. This alternative was added to judge the impact of no SINCGARS radios on 
the scenario and provide a measure of current architecture performance. 

b. Experimental Design 

A 2* factorial experimental design was selected. The name 2* relates 
to considering k factors each with two possible levels. This design allows the smallest 
number of treatment combinations with which k factors can be analyzed under a 


complete factorial arrangement[Ref.8]. 
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A 2* design may provide information on how sensitive the model’s 
output is to the different enhancements. This design can also provide insight into the 
interactions between the enhancements. The design is particularly well suited to this 
fidelity analysis in that the enhancements are inherently two level (on and off). 

The 2° design represents an experiment using three factors each with 
two levels. As stated earlier, the three factors analyzed were jammers, MTBF, and 
perishability. The incorporation of these three factors produced 8 possible treatment 
(upgrade) combinations. 


Table II Treatment (Upgrade) Combinations 


| code | description 
| 000 | Baseline Case allo 












Perishability only 
Jammers & Perishability 


7 MTBF & Perishability 
fs Topline Case (all on) 


The upgrade combinations (UC) are indexed with i which ranges from 1 to 8. The 





code represents the actual enhancements incorporated within each upgrade 
combination. The three digits of the code correspond to the three enhancements 
portrayed as perishability, MTBF and Jammers in that order. The digits 0 and 1 


correspond to off and on, respectively. 
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Each of the radio allocation alternatives were evaluated by MCCAAM 
under each of the 8 upgrade combinations. Each of these 24 (3 alternative x 8 upgrade 
combinations) model runs produced a steady state penalty plot that was analyzed using 
MCCAAM’s analysis routines. This analysis was performed on the penalty rates (R,,) 
generated by each model run. The initial condition analysis established the steady 
state point at 700 minutes into the 10,000 minute run. The remaining 9300 minutes 
were sampled at 25 minute intervals to produce approximately 360 samples. The batch 
size was set at 12 to produce 30 iid batch means or penalty rates. The autocorrelation 
of these batches was analyzed and found to be insignificant(max p = .023120). Each 
vector of penalty rates was then broken down into five groups of six samples and 
tabulated in matrices. 


Table III Sample R,,, Matrix 





Puce | samples (c=1...6) for Replication 
jAitemative {1 [2 [3 | 4 | 56 |6 | 
es Eee ee ee ee 
es ee ee ee ee ee ee 
Ps Ry Re | Re Ree | Re | Rass 











Rije = Penalty rate k of UCi and alternativej , k=1,...,30 


These penalty rates are then compared across the alternatives to 
determine the lowest penalty rate for each sample number. The variable Ti, is used 
to denote the alternative with the lowest (best) penalty rate. The winner receives a 


one while the remainder of the alternatives get zeros. 
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, If Ry S$ Rin, For all n¥j 


ijk = 1 < 
0, otherwise 


These values were then tabulated in the T;,, table and the group totals were calculated 


by summing across the samples. 


Table IV Sample T;, Matrix 





UC 2 Group 


| Altematve | 1 | 2 [3 | « | 5 [6 | OM 
te ea Oe ee) a 
ee eae ee ee ee ee ee ee 
ae a Se Da ee ee 










These group totals (A,,) reflect the number of times each alternative 
was selected as the best and range from zero to six. The group totals are used to 
calculate the Measure of Performance (MOP) values for each group. Once calculated, 


the group totals are tabulated in a group total matrix. 


Ay. = YS Tages L21,..,5 


1 Upgrade Combination (UC) index 
J = Alternative Number 
k Sample Number 


d = Group Number 
Ajj, = Number of Times Alternativej chosen for UC i in Groupl 
T,j;, = Lowest Penalty Rate for Sample k, loro 
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Table V Sample Group Total Matrix 





The measure of performance (MOP) for each upgrade combination is 
calculated by summing the products of weight and number of times chosen for each 
alternative. This sum is then divided by the group sample size to get the average 


value for each group. These 40 MOP values are then used to conduct an analysis of 


variance ANOVA. 
3 
ek - = 
MOR Ra Bani , for i=1,...,8, 1=1,...,5 
i = Upgrade Combination (UC) index 
j = Alternative Number 
1 = Group Number 
MOP,,= Measure of Performance for UC i 
Ajj, = Number of Times Alternativej chosen for UC i 
W; = Weight of Alternativej 


The weight for each alternative, W, was calculated using the topline case as outlined 
in section II] E 3. This weight represents the proportion of the time that an 
alternative was selected by the highest fidelity upgrade combination (the topline case). 
The group totals were summed for each alternative and then divided by the total 


number of samples (30) to determine each alternative’s weight. 
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Table VI Treatment (Upgrade) Combinations 


ie) 
3 
Q 
o 


Upgrade Combinations 
description 
ae h000n Baseline Case (all off) (1) 
Jammers only 


MTBF only 


4 Jammers & MTBF 
Perishability only 
Jammers & Perishability 
7 MTBF & Perishability 
2a 111 Topline Case (all on) 





f : 
ms |O 
Oo ie 
| 


— 
; 
_ 
Q f 


om 





Once all of the MOP values have been determined, multifactor analysis 
of variance (ANOVA) was utilized to calculate the relationship between the response 
variable (fidelity yield) and the three factors. The model’s accuracy depends on the 
assumption that the error terms are normally distributed and independent. The 


following linear equation was used to model this relationship. 
Viger = BT + BS tet (TB) 5+ (OY) set (BY) jet (COBY) ijn tE Giza 


jki= ith response with factors at levels i,j,k 


Y; 

ey = overall mean response 

t, = effect of Jammers at level i 

B,; = effect of MTBF at level j 

Y, = effect of Perishability at level k 


€ ijn. Che random error component 
ijk= factor level (1=o0n, 0=off) 
I = sample number (1l=1,...,6) 
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In this design, the factors t;, 8; and y, correspond to the three 
enhancements. The terms in parentheses indicate two-way (tf), and three-way (tBy)jx 
interactions of the corresponding factors. It is assumed that the random error terms, 
ga aFe independent, identically distributed normal variables with a mean of zero and 
a variance of o”[Ref. 8]. The treatment effects are defined as deviations from the 
overall mean so, 


2 
Da Or DBP Oe Dvn 0 


tel F=1 k=1 


Similarly, the interaction effects are fixed and defined so that they also sum to zero 


as shown below for the (tf), interaction. 


2 > 


B (TB) 45 = 2, (5B) 43 = 
=1 


j* 


In this factorial design, all three factors are of equal interest. We are specifically 


interested in testing hypotheses concerning the equality of the treatment effects. 


H,: 7, =T, = 0 


H,: at least one t, # 0 


We are also interested in the interactions between the treatments and therefore test 


each of the interaction terms. 


Fis (tB) ,; for all i,j 
H,: at re one (tB) ,; # 0 
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These hypotheses are tested using a multifactor ANOVA. The sum of 
squares for each treatment as well as the total sum of squares (SS) is calculated and 
divided by its degrees of freedom to obtain its mean square. The expected values of 


the mean squares (MS) are 


SS, 
ani a-1 





E(MS,) = E( 


a=b= 2 levels 
n= 5 groups of data 


If the null hypothesis for each treatment is true, then all of the expected mean 
squares estimate o”. However, if there are differences between treatment effects then 
that particular E(MS) value will be larger than the expected mean square error term 


E(MS,). 


E(MS,) = 0? 


Therefore, to test the significance of the main effects and their interactions, simply 
divide the corresponding mean square by the mean square error. Based on our 
assumption of independent identically distributed normal error terms with constant 
variance, o”, each of the ratios of mean squares are distributed as F with 1 degree of 
freedom in the numerator and 32 in the denominator. The critical region is then the 
upper tail of the F distribution. The procedure is summarized in the analysis of 


variance table in part d. of this subsection. 
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c. Weight Determination 
The weights were calculated by using the topline case to generate 
decision output that reflects the highest level of fidelity. This data was transformed 


into proportions that reflect the frequency that the alternative was chosen. 


Table VII Alternative Weights 





d. Results of the 2° Experiment 
The experiment was performed as discussed in the part b. The penalty 
rate (Rj,,) and T;,, matrices are consolidated in Appendix A. The Group Total matrices 
are also located in Appendix A. The MOP values (MOP,,) are displayed below. 
Table VIII MOP Values 
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These values were then analyzed using the multifactor ANOVA 
capabilities of Statgraphics version 5. The resulting ANOVA table revealed only one 
significant main effect and no significant interactions. 

Table IX ANOVA Results 
Analysis of Variance for Fidelity Yield 
i: 
Variation quare 
a 


ree 20736 
pic | 201 


.05041 


Coll 
cae 
EE 
ie 
Interactions [ae 
| ap | oni56 
fac 
ae ee 
| ape 
| Residual | a, 


.22801 1.834 


ere a 
Residual fase | oe | asso | 
[tot ssi [| 





The application of Jammers produced the only significant effect on the model’s fidelity 
yield(p = .015). The remaining two factors, MTBF and perishability produced roughly 
equivalent p values (.206 and .185 respectively) but were well above a conservative « 
value of .10. The interaction effects were even less significant with the exception of 
the BC term which corresponds to the interaction between MTBF and perishability. 
This term had a p value of .185 while the remainder were greater than 0.5. The 


interaction plots are in Appendix B. 
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F. STAGE 5 - FIDELITY DECISION 

The fidelity analysis demonstrates clearly that only the Jammer enhancement 
should be added to the upgraded model. Its significance to the model is highlighted 
by the results of the multifactor ANOVA as displayed in Table IX. The fidelity costs 
of the remaining enhancements greatly outweigh their impact on the model’s decision 
and should be omitted from the SINCGARS allocation determination process. A 
comprehensive record of these enhancements should be maintained however, because 


they may prove beneficial to a later upgrade effort or their present fidelity costs may 


be reduced by some new data source. 





V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

Computer simulation is poised on the threshold of an exciting new frontier. The 
continuing technological advances in computing power, both in hardware and software, 
have provided a software development environment that is conducive to model reuse. 
Object-Oriented simulation and open architecture systems are primary examples of 
advances that impact directly on model reuse. The diminishing requirements for 
model reimplementation will increase the availability of modelling effort for both new 
model development and the improvement of existing models. To properly navigate 
within this new frontier, a specialized methodology is required. 

The Fidelity Enhancement Process provides a useful map for conducting the 
upgrade of an existing model. Although the process is directed toward models 
incorporating OOS and open architecture, its five stages address all of the steps 
necessary for a successful model upgrade. The individual stages are general enough 
to allow their application to a wide range of decision making models. This 
methodology was tested during the upgrade of the Marine Corps Communication 
Architecture Analysis Model(MCCAAM). 

The Fidelity Enhancement Process received its initial application during the 
upgrade of MCCAAM. The stages proved beneficial in structuring the upgrade process 
to allow the rapid application of the required enhancements. The prototyping was very 
conducive to a group development effort in that the enhancements could be turned off 


to negate their impact on the remainder of the model. The fidelity analysis allowed 
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the developers to fine tune the final model and limit the fidelity costs. The 2" factoral 
design proved to be an effective technique for assessing the interactions between the 
enhancements as well as the main effects. The 2" factoral experimental design 
provides a solid foundation for the fidelity analysis stage. The key to the continued 
usefulness of the Fidelity Enhancement Process is the expansion of this stage by 


increasing the number of different analysis techniques used. 


B. RECOMMENDATIONS 

The tremendous potential of model reuse warrants continued emphasis. The 
Fidelity Enhancement Process should be applied to more models to validate its stages 
and expand the number of documented fidelity analysis techniques. A greater variety 
of well-documented analysis strategies and techniques will increase the usefulness of 
the process by providing more analysis options for its users. The additional 
applications may also uncover the need for modifications to the stages. 

The fidelity analysis stage presents the most potential for expansion or 
modification. Fidelity analysis is currently associated with the overall effect of an 
enhancement being turned on or off. Additional insight may be gained by conducting 
sensitivity analysis on an enhancement. This may alter the fidelity analysis or become 
part of the presentation of results. The actual incorporation of sensitivity analysis into 
the fidelity analysis is model dependent at this point, but any techniques utilized to 


address this issue will aid future users in tackling these problems. 
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APPENDIX A: DATA 







Upgrade Combination (1) code 000 





Alternative Grand Mean Standard Auto 
Deviation Correlation 
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1.66 
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UPGRADE COMBINATION 2 RAW DATA 


Upgrade Combination 2 code 001 
Alternative Grand Mean Standard Auto 
Deviation Correlation 


es ee ee ee 
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Upgrade Combination 2 code 001 


Samples 


ae re eae 
eee ee 


po Samptes 
| Alternative | 19 | 20 | 2 | 22 | 23 | a | 
. | 604 | 


ae 
ms 
=a 
eee 











Ty, TABLES 





Alternative 


ie 


Alternative 








46 








Ay, TABLE 





47 








ee ee Cae 


6.98 





48 











Upgrade Combination 3 code 010 










es ee 
| Alternative | 19 {| 14 | 15 | 16 | a7 | ie | 
a ae 


ees eRe eee 
| Alternative | 19 | 20 | 21 | 22 | 23 | 24 | 
| a | p05 | on | 599 | 578 | 27 | 608 | 
|| as | 567 | 573 | 498 | 205 | 7.04 
| 590 | 507 _| 5. . 














eS eet a ee Ue ee 
| Alternative | 25 | 26 | 27 | 28 | 20 | 30 | 
| | 4ze | 47 | 635 | 557 | 739 | 470 _| 
| 2 | 430 | 388 | 500 | 692 | 718 | 5.29 | 
ps | as | ase | sss | seo | 708 | 449 | 
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UPGRADE COMBINATION 4 RAW DATA 
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Upgrade Combination 5 code 100 













rs 

| Alternative | 19 | 20 | 21 | 2 | 23 | 24 
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UPGRADE COMBINATION 6 RAW DATA 
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| Upgrade Combination 6 code 101 | 
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| a | ao | 555 | ans | a4s | 296 | 548 
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UPGRADE COMBINATION 7 RAW DATA 


Upgrade Combination 7 code 110 
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UPGRADE COMBINATION 8 RAW DATA 


Upgrade Combination 8 code 111 
Alternative Grand Mean Standard Auto 
Deviation Correlation 
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Upgrade Combination 8 code 111 





| 13 

Pass [os [oe [am [om [a 
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| 413 | 460 | 400 | 641 | raz | 224 | 
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APPENDIX B: INTERACTION PLOTS 


Plot of Interactions for mermers by MTBF 
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Plot of Interactions for Jenner by 
fer lsreb! ity 


Perienabi tity 
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Piet of interactions for MTBF by 
Perisnmbi lity 


Pertenebi tity 
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